Skip to content

feat: offline update upload via web UI#1309

Open
agh wants to merge 6 commits into
jetkvm:devfrom
agh:feat/offline-updates
Open

feat: offline update upload via web UI#1309
agh wants to merge 6 commits into
jetkvm:devfrom
agh:feat/offline-updates

Conversation

@agh
Copy link
Copy Markdown
Contributor

@agh agh commented Mar 17, 2026

Implements the offline update feature requested in #96, following the design direction from #96 (comment).

Problem

Devices on air-gapped networks have no way to update firmware through the web UI. The only options are DFU mode or manual SSH — neither accessible to most users. The online update path requires both internet access (for downloading binaries) and keyserver access (for fetching the GPG signing key), making it unusable on isolated networks.

What this does

Adds an "Offline Update" section to Settings → General that lets users upload .tar.gz update archives directly through the web UI. No internet connection required on the device — not for the binary, not for the signing key.

Archive format

Each offline update archive contains four files:

jetkvm_app            # the update binary
jetkvm_app.sha256     # SHA256 hash (hex digest)
jetkvm_app.sig        # detached GPG signature
jetkvm_app.pub        # armored GPG public key used to produce the signature

System archives follow the same structure with update_system.tar as the binary name. The .pub file is what makes fully offline verification possible — the device validates it against the pinned root key fingerprint (internal/ota/gpg.go:21) before using it, the same trust anchor as the online path.

Signature verification

Verification is required and binary: it passes or it fails. There is no bypass option.

  1. Extract archive, validate structure (exactly 4 files, no path traversal)
  2. Compute SHA256 of the binary, compare to .sha256 — reject on mismatch
  3. Parse the bundled .pub, validate its fingerprint against the hardcoded root fingerprint via parseAndValidateKeyring — reject if it does not match
  4. Verify the .sig against the binary using the validated key — reject on failure

This is equivalent in security to the online path. The trust anchor is identical: the root key fingerprint compiled into the running firmware. A malicious archive cannot substitute a rogue key because the fingerprint check will reject it, whether the key arrives from a keyserver or from the archive.

Key rotation model

rootKeyFP is currently a single fingerprint. The rotation model for future key changes would be:

Release Trusts Signed with Can install offline archives signed by
v1 Key A Key A A
v2 Key A, Key B Key A A or B
v3 Key A, Key B Key B A or B
v4 Key B Key B B only

Rollback from v4 to v1 correctly fails — you cannot roll back past a key removal. Expanding rootKeyFP to []string is a separate change; not needed until a rotation is imminent.

Backend

Two HTTP endpoints behind protectedMiddleware:

  • POST /ota/upload — accepts a multipart .tar.gz archive with a component field (app or system). Extracts to a temp directory, runs the verification pipeline above, stages the verified binary at the standard OTA path on success.

  • POST /ota/apply — applies a previously verified and staged offline update. Triggers rk_ota for system updates or binary swap for app updates. The device reboots on completion.

VerifySignatureFromFileWithKey is a new method on GPGVerifier that accepts raw armored key bytes, validates the fingerprint via parseAndValidateKeyring, and calls CheckDetachedSignature with the resulting single-entity keyring. No keyserver interaction, no cache mutation. Depends on the parseAndValidateKeyring fix from #1316.

The existing applySystemImage() (extracted from updateSystem()) is reused for system updates.

Frontend

OfflineUpdateCard component with two independent sections (app / system). Each has a file input, upload progress bar, and verification status indicators showing hash and signature results. Rendered on the general settings page between the auto-update toggle and the reboot button.

Release pipeline

offline_archive_app Makefile target packages jetkvm_app + .sha256 + .sig + .pub into jetkvm_app_offline_update.tar.gz. Wired into the production release flow.

The equivalent system archive target is not included — the system image is built in a separate repo that does not have GPG signing in its pipeline yet.

Tests

31 tests in internal/ota/offline_test.go:

Extraction (10): valid app/system, missing hash/sig/pub/binary, unexpected files, path traversal, corrupt gzip, nested directories

Verification (8): valid signature, hash mismatch, invalid signature, wrong key (fingerprint mismatch on bundled .pub), empty signature, empty public key, truncated signature, corrupted binary on disk

End-to-end (5): full extract→verify pipeline — valid archive, tampered binary, wrong signer, wrong hash, system component

Utilities (3): ComponentUpdatePath (app/system/unknown)

Other (5): hashFile, VerifySignatureFromFileWithKey (covered transitively through offline tests and TestParseAndValidateKeyring_FiltersRogueKeys in gpg_test.go)

Not included

  • E2E Playwright tests (want feedback on the approach first)
  • Drag-and-drop on the file input
  • offline_archive_system Makefile target (blocked on system repo signing pipeline)
  • rootKeyFP[]string expansion (separate change)

Closes #96

@agh agh mentioned this pull request Mar 17, 2026
@equinox0815
Copy link
Copy Markdown

hmmm, now that i have looked deeper into this and #1188 my question from earlier is answered.

Boy-oh-boy i have to say that i find it very hard to express my bewilderment about how the signature mechanism works without using foul language. Why is this so horrendously complicated? Downloading the singing key from a keyserver is just something i have never seen any software do. And sorry i don't mean that in a good way. This dependency on a keyserver being reachable in order to upgrade the device makes the whole offline upgrade feature moot. How many devices out there will be unable to download the binaries directly but will be able to reach a keyserver?
What were the design goals in mind when the signature verification was implemented? Usually adding this much unneeded complexity is not the path to a robust and secure mechanism. I strongly encourage you to revisit the design decisions that yielded this result. All i can say is that now i am even more convinced than ever that it was a good idea to put this device in my management VLAN without any internet access.

Of course, this is not your fault @agh. This is also likely not the place to discuss this but #1188 has already been merged so apparently the project team does not see it my way. I guess i have to live with that. The only thing i can think of to make upgrades truly offline would be to also add a keyring containing the signing key to the tarball and importing it as if it would come from a keyserver. Of course this makes this setup even more complex but at least it makes sense this way.

parseAndValidateKeyring validates that at least one entity in a
fetched keyring matches the pinned root key fingerprint
(rootKeyFingerprint, gpg.go:21). On match, it returns the entire
keyring — including any additional entities the keyserver included
in its response.

This is a problem because openpgp.CheckDetachedSignature iterates
every key in the provided keyring and accepts a signature from any
of them. A compromised or malicious keyserver could return a
response containing the legitimate JetKVM release key (satisfying
the fingerprint check) alongside an attacker-controlled key. A
binary signed with the attacker key would then pass verification
in both VerifySignature and VerifySignatureFromFile, since both
pass the cached keyring directly to CheckDetachedSignature.

The fix is a single-line change: return openpgp.EntityList{entity}
instead of the full keyring when the fingerprint matches. This
ensures only the trusted key is ever used for signature verification
regardless of what a keyserver returns.

TestParseAndValidateKeyring_FiltersRogueKeys exercises this by
constructing a two-entity armored keyring (trusted + rogue),
passing it through parseAndValidateKeyring, asserting the returned
keyring contains exactly one entity with the correct fingerprint,
and confirming that CheckDetachedSignature rejects a signature
produced by the rogue key.

Reported-by: equinox0815

Signed-off-by: Alex Howells <alex@howells.me>
@agh agh marked this pull request as draft March 22, 2026 22:31
@agh agh changed the title WIP: feat: offline update upload via web UI feat: offline update upload via web UI Mar 22, 2026
agh added 4 commits March 22, 2026 15:39
Introduce internal/ota/offline.go with two primary functions:

- ExtractOfflineArchive() reads a .tar.gz upload, validates that it
  contains exactly the expected files (binary + .sha256 + .sig),
  rejects path traversal and unexpected entries, strips leading
  directory prefixes (for archives created with tar czf wrapper dirs),
  and returns an OfflineBundle struct.

- VerifyOfflineBundle() runs SHA256 verification against the hash
  from the archive (hard reject on mismatch), then attempts GPG
  signature verification via the existing GPGVerifier. When keyservers
  are unreachable (air-gapped device), returns KeyFetchFailed=true
  instead of rejecting, allowing the caller to prompt the user for
  bypass confirmation. Bad signatures (key available, sig invalid)
  are always fatal.

Refactor updateSystem() to extract applySystemImage() as a reusable
function for running rk_ota on a staged system tar. Add
ApplyOfflineUpdate() to State for the offline apply flow, plus
GPGVerifier() and ComponentUpdatePath() accessors.

Table-driven tests in offline_test.go cover extraction (valid
app/system archives, missing hash, missing sig, missing binary,
unexpected files, path traversal, corrupt gzip, nested directories)
and verification (valid signature, hash mismatch, invalid signature,
wrong signing key, empty signature, key fetch failure for air-gapped
devices, truncated signature, corrupted binary on disk).

Signed-off-by: Alex Howells <alex@howells.me>
Introduce two new HTTP endpoints behind protectedMiddleware:

- POST /ota/upload: accepts multipart form with a .tar.gz offline
  update archive and a component field (app or system). Extracts
  the archive to a temp directory, validates structure via
  ExtractOfflineArchive(), runs hash and GPG verification via
  VerifyOfflineBundle(), and stages the verified binary at the
  standard OTA path (/userdata/jetkvm/jetkvm_app.update or
  update_system.tar). Returns a JSON response indicating hash,
  signature, and key-fetch status so the frontend can prompt for
  signature bypass on air-gapped devices. Enforces a 200MB upload
  limit and rejects requests when an update is already in progress.

- POST /ota/apply: accepts JSON with component and bypassSignature
  fields. Verifies a staged file exists, then delegates to
  ApplyOfflineUpdate() which runs rk_ota for system images or
  triggers reboot for app binaries. Disables auto-update since
  the user explicitly chose a version. The apply runs asynchronously
  and the device reboots on completion.

The handlers live in ota_offline.go to keep web.go focused on
routing. Routes registered in web.go protected group alongside
the existing /storage/upload endpoint.

Signed-off-by: Alex Howells <alex@howells.me>
Add OfflineUpdateCard component with two independent file upload
sections (app and system). Each section provides:

- File input accepting .tar.gz archives
- Upload progress bar using XMLHttpRequest progress events
- Verification status display (hash ✓, signature ✓)
- Signature bypass prompt when the device cannot reach GPG
  keyservers (air-gapped networks): amber warning card explaining
  the situation with explicit 'Apply Without Signature Verification'
  confirmation
- Error display with retry option

The component communicates with POST /ota/upload for the upload+verify
phase and POST /ota/apply for the apply+reboot phase.

Rendered on the general settings page between auto-update toggle and
reboot, making offline updates discoverable without cluttering the
existing online update dialog flow.

Adds 12 i18n keys to en.json and regenerates paraglide output.

Signed-off-by: Alex Howells <alex@howells.me>
Add offline_archive_app Makefile target that packages jetkvm_app,
jetkvm_app.sha256, and jetkvm_app.sig into a single
jetkvm_app_offline_update.tar.gz archive. The archive is the
upload format expected by POST /ota/upload.

Wire the target into the production release flow: after signing
and uploading individual artefacts to Cloudflare R2 (the existing
OTA update hosting bucket), the offline archive is built and
uploaded alongside them. The GitHub Release draft now includes
the offline archive as an additional download.

The system offline archive target is deferred until the system
repo has GPG signing in its build pipeline.

Signed-off-by: Alex Howells <alex@howells.me>
@agh agh force-pushed the feat/offline-updates branch from ce41145 to a770adf Compare March 22, 2026 23:07
The offline update path previously fell back to a "warn and bypass"
prompt when the device could not reach GPG keyservers to fetch the
signing key. This defeated the purpose of offline updates, which
exist precisely for air-gapped devices without internet access.

The archive format now includes a .pub file (armored GPG public key)
alongside the binary, .sha256, and .sig. On upload, the bundled key
is validated against the pinned root fingerprint in gpg.go:21 via
parseAndValidateKeyring — the same trust anchor used by the online
update path. If the fingerprint matches, the key is used to verify
the signature locally. No keyserver call is made.

Verification is now binary: it passes or it fails. There is no
third "key fetch failed" state and no bypass option.

Backend (internal/ota/offline.go, ota_offline.go):
- OfflineBundle gains PublicKeyData []byte field
- ExtractOfflineArchive requires .pub (4 files, up from 3)
- VerifyOfflineBundle calls new VerifySignatureFromFileWithKey
  method on GPGVerifier instead of VerifySignatureFromFile
- Removed isKeyFetchError(), KeyFetchFailed from OfflineVerifyResult,
  BypassSignature from offlineUpdateApplyRequest, offlineUploadTimeout
- VerifyOfflineBundle no longer takes a context parameter

New GPG method (internal/ota/gpg.go):
- VerifySignatureFromFileWithKey accepts raw armored key bytes,
  validates the fingerprint via parseAndValidateKeyring, then
  calls CheckDetachedSignature with the resulting single-entity
  keyring. No keyserver interaction, no cache mutation.

Frontend (ui/src/components/OfflineUpdateCard.tsx):
- Removed bypass confirmation dialog, showBypassPrompt state,
  keyFetchFailed from UploadResult, bypassSignature from apply
  request, ExclamationTriangleIcon import
- Signature OK indicator now always shows on verified state

Localisation (ui/localization/messages/en.json):
- Removed offline_update_signature_bypass_title,
  offline_update_signature_bypass_description,
  offline_update_signature_bypass_confirm

Tests (internal/ota/offline_test.go):
- All archives now include .pub files
- newOfflineSigningFixture replaces newSigningTestFixture for
  offline tests — no mock HTTP client needed
- Added TestExtractOfflineArchive_MissingPub,
  TestVerifyOfflineBundle_EmptyPublicKey,
  TestVerifyOfflineBundle_WrongKey (bundled key fingerprint
  mismatch)
- Removed TestIsKeyFetchError,
  TestVerifyOfflineBundle_KeyFetchFailure
- End-to-end tests updated for 4-file archive format

Makefile:
- offline_archive_app target includes jetkvm_app.pub

Key rotation is not addressed here. rootKeyFP is a single string;
expanding to []string for the v1→v2→v3→v4 key rollover model is
a separate change.

Signed-off-by: Alex Howells <alex@howells.me>
@agh agh force-pushed the feat/offline-updates branch from a770adf to cf6e308 Compare March 22, 2026 23:16
@agh agh marked this pull request as ready for review March 22, 2026 23:17
@agh
Copy link
Copy Markdown
Contributor Author

agh commented Mar 22, 2026

Okay, @adamshiervani, I think this is probably worth reviewing now. 👍🏻

@Gunni
Copy link
Copy Markdown

Gunni commented Mar 22, 2026

Remove dependency on keyserver, unless you already did, it makes the whole feature moot.

My device will never have internet access.

terryrankine added a commit to terryrankine/kvm that referenced this pull request May 7, 2026
Implements POST /ota/upload and POST /ota/apply endpoints for uploading
a .tar.gz archive containing a signed binary and SHA-256 hash file.
The handler extracts, verifies the SHA-256 digest, stages the binary at
/userdata/picokvm/bin/kvm_app or /userdata/picokvm/update_system.tar,
then reboots on apply.

Adds OfflineUpdateCard UI component to the Version settings panel,
surfacing app and system update slots with upload progress, verification
status, and one-click apply.

Adapted from upstream jetkvm#1309: replaced internal/ota package
references with fork's flat architecture; removed GPG verification
(fork has no trust-anchor infrastructure); adjusted storage paths from
/userdata/jetkvm/ to /userdata/picokvm/.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Offline updates

3 participants